Jupyter at Bryn Mawr College
Public notebooks: /services/public/dblank / jupyter.cs / Examples

1. The Multiple Roles of Anticipation in Developmental Robotics

Replicating the experiments in https://www.aaai.org/Papers/Symposia/Fall/2005/FS-05-05/FS05-05-002.pdf

First, we import conx and create a 6-10-2+2 feed-forward network:

In [2]:
from calysto.ai.conx import Network
import random
In [3]:
class XorNetwork(Network):
    def __init__(self):
        super().__init__()
        self.totalCorrect = 0
        self.totalPossible = 0
        self.totalTSSError = 0
        
    def correct(self, layername):
        c = 0
        for i in range(len(self[layername].activation)):
            if abs(self[layername].activation[i] - self[layername].target[i]) < self.tolerance:
                c += 1
        return c

    def reportPattern(self):
        if (self["input"].activation[0] == 1 and
            self["input"].activation[1] == 1):
            c = self.correct("output")
            self.totalCorrect += c
            self.totalPossible += 2
            self.totalTSSError += self["output"].TSSError()
            
    def reportEpoch(self, *args):
        print("Epoch:", self.epoch, end=" ")
        print("Percent Correct:", self.totalCorrect/float(self.totalPossible), end=" ")
        print("TSS Error:", self.totalTSSError)
        self.totalCorrect = 0
        self.totalPossible = 0
        self.totalTSSError = 0
In [4]:
net = XorNetwork()

Conx using seed: 1437735633.2086067
In [5]:
net.addLayer("input", 6)
net.addLayer("hidden", 10)
net.addLayer("output", 2)
net.addLayer("hints", 2)

net.connect("input", "hidden")
net.connect("hidden", "output")
net.connect("hidden", "hints")
In [6]:
net.setTolerance(.4) # outputs must be within this of targets to consider correct
net.setMomentum(0.0)
net.setEpsilon(.6)
net.resetCount = 1
net.resetEpoch = 1000
net.setStopPercent(1.1) # don't stop

We write a couple of helper functions to generate the inputs and targets:

In [7]:
def bin2list(n, width):
    return [int(b) for b in (("%0" + str(width) + "d") % int(bin(n)[2:]))]

Test bin2list:

In [8]:
bin2list(1, 6)
Out[8]:
[0, 0, 0, 0, 0, 1]
In [9]:
def xor(a, b):
    return a ^ b

Test xor:

In [10]:
for group in range(4):
    list_group = bin2list(group, 2)
    print(list_group, ":", xor(*list_group))

[0, 0] : 0
[0, 1] : 1
[1, 0] : 1
[1, 1] : 0
In [11]:
inputs = []
targets = []
for group in range(4):
    list_group = bin2list(group, 2)
    for i in range(2 ** 4):
        list_i = bin2list(i, 4)
        inputs.append(list_group +  list_i)
        targets.append([xor(*list_i[0:2]), xor(*list_i[2:4])] + [0.5, 0.5])
In [12]:
len(inputs), len(targets)
Out[12]:
(64, 64)
In [13]:
net.mapTarget("output", 0)
net.mapTarget("hints", 2)

Now we are ready to set the inputs and targets and train:

In [14]:
net.setInputs(inputs)
In [15]:
net.setTargets(targets)

1.1 Standard XOR Problem

In [16]:
net.initialize()

Initializing 'Backprop Network' weights...
In [17]:
%%time
net.train()

Epoch: 25 Percent Correct: 0.085 TSS Error: 210.387959588
Epoch: 50 Percent Correct: 0.0825 TSS Error: 206.114479848
Epoch: 75 Percent Correct: 0.42375 TSS Error: 172.805874726
Epoch: 100 Percent Correct: 0.48125 TSS Error: 161.506952252
Epoch: 125 Percent Correct: 0.9725 TSS Error: 20.4823148419
Epoch: 150 Percent Correct: 1.0 TSS Error: 1.61885719386
Epoch: 175 Percent Correct: 1.0 TSS Error: 0.635201165256
Epoch: 200 Percent Correct: 1.0 TSS Error: 0.347357788025
Epoch: 225 Percent Correct: 1.0 TSS Error: 0.224530690366
Epoch: 250 Percent Correct: 1.0 TSS Error: 0.157763387438
Epoch: 275 Percent Correct: 1.0 TSS Error: 0.116869609826
Epoch: 300 Percent Correct: 1.0 TSS Error: 0.0906929986352
Epoch: 325 Percent Correct: 1.0 TSS Error: 0.0724010081715
Epoch: 350 Percent Correct: 1.0 TSS Error: 0.0594036871919
Epoch: 375 Percent Correct: 1.0 TSS Error: 0.0494409150958
Epoch: 400 Percent Correct: 1.0 TSS Error: 0.0419186327269
Epoch: 425 Percent Correct: 1.0 TSS Error: 0.0360186050152
Epoch: 450 Percent Correct: 1.0 TSS Error: 0.0312728817568
Epoch: 475 Percent Correct: 1.0 TSS Error: 0.0274384030386
Epoch: 500 Percent Correct: 1.0 TSS Error: 0.024267839265
Epoch: 525 Percent Correct: 1.0 TSS Error: 0.0216279424131
Epoch: 550 Percent Correct: 1.0 TSS Error: 0.019398309974
Epoch: 575 Percent Correct: 1.0 TSS Error: 0.017496471468
Epoch: 600 Percent Correct: 1.0 TSS Error: 0.0158691034136
Epoch: 625 Percent Correct: 1.0 TSS Error: 0.0144541750188
Epoch: 650 Percent Correct: 1.0 TSS Error: 0.0132248510641
Epoch: 675 Percent Correct: 1.0 TSS Error: 0.0121386998779
Epoch: 700 Percent Correct: 1.0 TSS Error: 0.0112001659546
Epoch: 725 Percent Correct: 1.0 TSS Error: 0.0103599951082
Epoch: 750 Percent Correct: 1.0 TSS Error: 0.00960425529018
Epoch: 775 Percent Correct: 1.0 TSS Error: 0.00893513030208
Epoch: 800 Percent Correct: 1.0 TSS Error: 0.00833198497809
Epoch: 825 Percent Correct: 1.0 TSS Error: 0.00778593751153
Epoch: 850 Percent Correct: 1.0 TSS Error: 0.00729620767092
Epoch: 875 Percent Correct: 1.0 TSS Error: 0.00685148658247
Epoch: 900 Percent Correct: 1.0 TSS Error: 0.0064473166284
Epoch: 925 Percent Correct: 1.0 TSS Error: 0.00607469823391
Epoch: 950 Percent Correct: 1.0 TSS Error: 0.00573633031302
Epoch: 975 Percent Correct: 1.0 TSS Error: 0.0054244256803
Epoch: 1000 Percent Correct: 1.0 TSS Error: 0.00513802981646
Reset limit reached; ending without reaching goal
Final #  1000 | TSS Error: 0.0008 | Correct: 1.0000 | RMS Error: 0.0018
   Final #  1000, Layer = 'hints'      | Units: 1.0000 | Patterns: 1.0000
   Final #  1000, Layer = 'output'     | Units: 1.0000 | Patterns: 1.0000
CPU times: user 53.9 s, sys: 0 ns, total: 53.9 s
Wall time: 54.1 s

Learns the patterns very quickly, usually in less that 100 epochs.

In [17]:
net.interactive = 1
net.sweep()
net.interactive = 0

-----------------------------------Pattern # 49
Display network 'Backprop Network':
=============================
Layer 'hints': (Kind: Output, Size: 2, Active: 1, Frozen: 0)
Target    :  0.50 0.50 
Activation:  0.50 0.50 
=============================
Layer 'output': (Kind: Output, Size: 2, Active: 1, Frozen: 0)
Target    :  0.00 1.00 
Activation:  0.00 1.00 
=============================
Layer 'hidden': (Kind: Hidden, Size: 10, Active: 1, Frozen: 0)
Activation:  1.00 0.06 0.23 0.03 0.05 0.17 0.96 0.16 0.10 0.00 
=============================
Layer 'input': (Kind: Input, Size: 6, Active: 1, Frozen: 0)
Activation:  1.00 1.00 0.00 0.00 0.00 1.00 
<q>uit, <g>o, or press Enter: 
-----------------------------------Pattern # 20
Display network 'Backprop Network':
=============================
Layer 'hints': (Kind: Output, Size: 2, Active: 1, Frozen: 0)
Target    :  0.50 0.50 
Activation:  0.50 0.50 
=============================
Layer 'output': (Kind: Output, Size: 2, Active: 1, Frozen: 0)
Target    :  0.00 0.00 
Activation:  0.00 0.00 
=============================
Layer 'hidden': (Kind: Hidden, Size: 10, Active: 1, Frozen: 0)
Activation:  0.13 0.04 0.05 0.90 0.00 0.06 0.00 0.04 0.02 0.89 
=============================
Layer 'input': (Kind: Input, Size: 6, Active: 1, Frozen: 0)
Activation:  1.00 1.00 1.00 1.00 0.00 0.00 
<q>uit, <g>o, or press Enter: g

1.2 XOR in Noisy Environment

Now, we will make the task a bit harder by corrupting some patterns so that they cannot be learned:

In [27]:
def postPropagate(*args, **kwargs):
    input_pattern = kwargs["input"]
    if input_pattern[:2] in [[0, 0], [0, 1], [1, 0]]:
        for i in range(2):
            kwargs["output"][i] = random.random() #random.randint(0, 1)
    target_pattern = kwargs["hints"]
    for i in range(2):
        error = net["output"][i].activation - kwargs["output"][i]
        target_pattern[i] = abs(error) # max(min((error + 1.0)/2.0, 1.0), 0.0)
    return kwargs

net.postPropagate = postPropagate

If an input starts with [0,0], [0,1] or [1,0] then we randomly change the targets.

In [28]:
net.initialize()

Initializing 'Backprop Network' weights...
In [29]:
%%time
net.train()

Epoch: 25 Percent Correct: 0.48165760869565216 TSS Error: 207.246010391
Epoch: 50 Percent Correct: 0.015 TSS Error: 207.664382012
Epoch: 75 Percent Correct: 0.0225 TSS Error: 207.731268183
Epoch: 100 Percent Correct: 0.01 TSS Error: 207.819999006
Epoch: 125 Percent Correct: 0.025 TSS Error: 206.749954314
Epoch: 150 Percent Correct: 0.0175 TSS Error: 206.341989881
Epoch: 175 Percent Correct: 0.0175 TSS Error: 207.682095538
Epoch: 200 Percent Correct: 0.0175 TSS Error: 205.65654072
Epoch: 225 Percent Correct: 0.02 TSS Error: 206.406838443
Epoch: 250 Percent Correct: 0.005 TSS Error: 207.952140679
Epoch: 275 Percent Correct: 0.01 TSS Error: 206.11341432
Epoch: 300 Percent Correct: 0.0175 TSS Error: 204.903412577
Epoch: 325 Percent Correct: 0.015 TSS Error: 207.163058142
Epoch: 350 Percent Correct: 0.01375 TSS Error: 205.657379616
Epoch: 375 Percent Correct: 0.02625 TSS Error: 207.758648002
Epoch: 400 Percent Correct: 0.01375 TSS Error: 207.236668269
Epoch: 425 Percent Correct: 0.0125 TSS Error: 207.235352555
Epoch: 450 Percent Correct: 0.025 TSS Error: 206.035634172
Epoch: 475 Percent Correct: 0.015 TSS Error: 206.760542495
Epoch: 500 Percent Correct: 0.0125 TSS Error: 206.189978998
Epoch: 525 Percent Correct: 0.015 TSS Error: 207.130356074
Epoch: 550 Percent Correct: 0.01 TSS Error: 206.163549657
Epoch: 575 Percent Correct: 0.01 TSS Error: 208.295400147
Epoch: 600 Percent Correct: 0.0225 TSS Error: 206.767899701
Epoch: 625 Percent Correct: 0.00875 TSS Error: 206.346999274
Epoch: 650 Percent Correct: 0.0275 TSS Error: 205.569044284
Epoch: 675 Percent Correct: 0.005 TSS Error: 206.188091389
Epoch: 700 Percent Correct: 0.00875 TSS Error: 206.944662583
Epoch: 725 Percent Correct: 0.0275 TSS Error: 206.146348102
Epoch: 750 Percent Correct: 0.01875 TSS Error: 205.71528004
Epoch: 775 Percent Correct: 0.0175 TSS Error: 205.052459473
Epoch: 800 Percent Correct: 0.01875 TSS Error: 205.867722752
Epoch: 825 Percent Correct: 0.0175 TSS Error: 208.033197286
Epoch: 850 Percent Correct: 0.015 TSS Error: 207.366018869
Epoch: 875 Percent Correct: 0.0075 TSS Error: 206.100793193
Epoch: 900 Percent Correct: 0.0175 TSS Error: 206.240182651
Epoch: 925 Percent Correct: 0.02125 TSS Error: 206.969226552
Epoch: 950 Percent Correct: 0.0125 TSS Error: 206.842018432
Epoch: 975 Percent Correct: 0.025 TSS Error: 205.948221626
Epoch: 1000 Percent Correct: 0.02 TSS Error: 205.373884697
Reset limit reached; ending without reaching goal
Final #  1000 | TSS Error: 18.5024 | Correct: 0.8047 | RMS Error: 0.2688
   Final #  1000, Layer = 'hints'      | Units: 1.0000 | Patterns: 1.0000
   Final #  1000, Layer = 'output'     | Units: 0.6094 | Patterns: 0.4531
CPU times: user 55.6 s, sys: 3.91 ms, total: 55.6 s
Wall time: 55.6 s

Standard backprop cannot learn the pattern in over 3k+ epochs.

In [ ]:
net.interactive = 1
net.sweep()
net.interactive = 0

-----------------------------------Pattern # 44
Display network 'Backprop Network':
=============================
Layer 'hints': (Kind: Output, Size: 2, Active: 1, Frozen: 0)
Target    :  0.55 0.37 
Activation:  0.51 0.48 
=============================
Layer 'output': (Kind: Output, Size: 2, Active: 1, Frozen: 0)
Target    :  0.00 0.00 
Activation:  0.55 0.37 
=============================
Layer 'hidden': (Kind: Hidden, Size: 20, Active: 1, Frozen: 0)
Activation:  0.02 0.03 0.02 0.02 0.02 0.02 0.02 0.02 0.23 0.01 0.02 0.02 0.02 0.02 0.02 0.04 0.02 0.02 0.02 0.02 
=============================
Layer 'input': (Kind: Input, Size: 6, Active: 1, Frozen: 0)
Activation:  1.00 0.00 1.00 0.00 0.00 0.00 
<q>uit, <g>o, or press Enter: 
-----------------------------------Pattern # 57
Display network 'Backprop Network':
=============================
Layer 'hints': (Kind: Output, Size: 2, Active: 1, Frozen: 0)
Target    :  0.48 0.28 
Activation:  0.51 0.45 
=============================
Layer 'output': (Kind: Output, Size: 2, Active: 1, Frozen: 0)
Target    :  1.00 0.00 
Activation:  0.52 0.28 
=============================
Layer 'hidden': (Kind: Hidden, Size: 20, Active: 1, Frozen: 0)
Activation:  0.01 0.00 0.01 0.01 0.01 0.01 0.02 0.01 0.41 0.01 0.01 0.01 0.01 0.01 0.01 0.01 0.01 0.01 0.00 0.02 
=============================
Layer 'input': (Kind: Input, Size: 6, Active: 1, Frozen: 0)
Activation:  1.00 1.00 0.00 1.00 0.00 0.00 
<q>uit, <g>o, or press Enter: 
-----------------------------------Pattern # 29
Display network 'Backprop Network':
=============================
Layer 'hints': (Kind: Output, Size: 2, Active: 1, Frozen: 0)
Target    :  0.46 0.47 
Activation:  0.51 0.50 
=============================
Layer 'output': (Kind: Output, Size: 2, Active: 1, Frozen: 0)
Target    :  1.00 0.00 
Activation:  0.54 0.47 
=============================
Layer 'hidden': (Kind: Hidden, Size: 20, Active: 1, Frozen: 0)
Activation:  0.02 0.02 0.02 0.02 0.02 0.02 0.02 0.02 0.01 0.02 0.02 0.02 0.02 0.02 0.02 0.03 0.02 0.02 0.02 0.02 
=============================
Layer 'input': (Kind: Input, Size: 6, Active: 1, Frozen: 0)
Activation:  0.00 0.00 0.00 1.00 0.00 1.00 
<q>uit, <g>o, or press Enter: 
-----------------------------------Pattern # 28
Display network 'Backprop Network':
=============================
Layer 'hints': (Kind: Output, Size: 2, Active: 1, Frozen: 0)
Target    :  0.44 0.47 
Activation:  0.51 0.50 
=============================
Layer 'output': (Kind: Output, Size: 2, Active: 1, Frozen: 0)
Target    :  1.00 0.00 
Activation:  0.56 0.47 
=============================
Layer 'hidden': (Kind: Hidden, Size: 20, Active: 1, Frozen: 0)
Activation:  0.00 0.00 0.00 0.00 0.00 0.00 0.01 0.00 0.01 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.01 
=============================
Layer 'input': (Kind: Input, Size: 6, Active: 1, Frozen: 0)
Activation:  1.00 1.00 1.00 0.00 1.00 1.00 
<q>uit, <g>o, or press Enter: 
-----------------------------------Pattern # 27
Display network 'Backprop Network':
=============================
Layer 'hints': (Kind: Output, Size: 2, Active: 1, Frozen: 0)
Target    :  0.58 0.57 
Activation:  0.50 0.49 
=============================
Layer 'output': (Kind: Output, Size: 2, Active: 1, Frozen: 0)
Target    :  0.00 1.00 
Activation:  0.58 0.43 
=============================
Layer 'hidden': (Kind: Hidden, Size: 20, Active: 1, Frozen: 0)
Activation:  0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.04 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 
=============================
Layer 'input': (Kind: Input, Size: 6, Active: 1, Frozen: 0)
Activation:  1.00 1.00 1.00 1.00 0.00 1.00 
<q>uit, <g>o, or press Enter: 
-----------------------------------Pattern # 22
Display network 'Backprop Network':
=============================
Layer 'hints': (Kind: Output, Size: 2, Active: 1, Frozen: 0)
Target    :  0.55 0.40 
Activation:  0.51 0.48 
=============================
Layer 'output': (Kind: Output, Size: 2, Active: 1, Frozen: 0)
Target    :  0.00 0.00 
Activation:  0.55 0.40 
=============================
Layer 'hidden': (Kind: Hidden, Size: 20, Active: 1, Frozen: 0)
Activation:  0.01 0.01 0.01 0.01 0.01 0.01 0.01 0.01 0.13 0.01 0.01 0.01 0.01 0.01 0.01 0.02 0.01 0.01 0.01 0.01 
=============================
Layer 'input': (Kind: Input, Size: 6, Active: 1, Frozen: 0)
Activation:  0.00 1.00 1.00 1.00 0.00 0.00 
<q>uit, <g>o, or press Enter: 
-----------------------------------Pattern # 51
Display network 'Backprop Network':
=============================
Layer 'hints': (Kind: Output, Size: 2, Active: 1, Frozen: 0)
Target    :  0.54 0.44 
Activation:  0.51 0.50 
=============================
Layer 'output': (Kind: Output, Size: 2, Active: 1, Frozen: 0)
Target    :  0.00 0.00 
Activation:  0.54 0.44 
=============================
Layer 'hidden': (Kind: Hidden, Size: 20, Active: 1, Frozen: 0)
Activation:  0.00 0.01 0.00 0.00 0.00 0.00 0.01 0.00 0.03 0.00 0.00 0.00 0.00 0.00 0.00 0.01 0.00 0.00 0.00 0.00 
=============================
Layer 'input': (Kind: Input, Size: 6, Active: 1, Frozen: 0)
Activation:  1.00 0.00 1.00 1.00 0.00 1.00 
<q>uit, <g>o, or press Enter: 
-----------------------------------Pattern # 36
Display network 'Backprop Network':
=============================
Layer 'hints': (Kind: Output, Size: 2, Active: 1, Frozen: 0)
Target    :  0.50 0.60 
Activation:  0.51 0.49 
=============================
Layer 'output': (Kind: Output, Size: 2, Active: 1, Frozen: 0)
Target    :  0.00 1.00 
Activation:  0.50 0.40 
=============================
Layer 'hidden': (Kind: Hidden, Size: 20, Active: 1, Frozen: 0)
Activation:  0.02 0.02 0.02 0.02 0.01 0.02 0.04 0.02 0.07 0.02 0.02 0.03 0.01 0.01 0.03 0.03 0.02 0.02 0.01 0.05 
=============================
Layer 'input': (Kind: Input, Size: 6, Active: 1, Frozen: 0)
Activation:  1.00 0.00 0.00 0.00 1.00 0.00 
<q>uit, <g>o, or press Enter: 
-----------------------------------Pattern # 55
Display network 'Backprop Network':
=============================
Layer 'hints': (Kind: Output, Size: 2, Active: 1, Frozen: 0)
Target    :  0.50 0.46 
Activation:  0.51 0.50 
=============================
Layer 'output': (Kind: Output, Size: 2, Active: 1, Frozen: 0)
Target    :  1.00 0.00 
Activation:  0.50 0.46 
=============================
Layer 'hidden': (Kind: Hidden, Size: 20, Active: 1, Frozen: 0)
Activation:  0.01 0.01 0.01 0.01 0.01 0.01 0.01 0.01 0.01 0.01 0.01 0.01 0.01 0.01 0.01 0.02 0.01 0.01 0.01 0.01 
=============================
Layer 'input': (Kind: Input, Size: 6, Active: 1, Frozen: 0)
Activation:  0.00 0.00 1.00 1.00 0.00 1.00 
<q>uit, <g>o, or press Enter: 
-----------------------------------Pattern # 34
Display network 'Backprop Network':
=============================
Layer 'hints': (Kind: Output, Size: 2, Active: 1, Frozen: 0)
Target    :  0.52 0.74 
Activation:  0.51 0.45 
=============================
Layer 'output': (Kind: Output, Size: 2, Active: 1, Frozen: 0)
Target    :  0.00 1.00 
Activation:  0.52 0.26 
=============================
Layer 'hidden': (Kind: Hidden, Size: 20, Active: 1, Frozen: 0)
Activation:  0.02 0.02 0.02 0.02 0.02 0.02 0.02 0.02 0.36 0.02 0.02 0.02 0.02 0.02 0.02 0.04 0.02 0.02 0.02 0.03 
=============================
Layer 'input': (Kind: Input, Size: 6, Active: 1, Frozen: 0)
Activation:  1.00 0.00 0.00 1.00 0.00 0.00 
<q>uit, <g>o, or press Enter: 
-----------------------------------Pattern # 11
Display network 'Backprop Network':
=============================
Layer 'hints': (Kind: Output, Size: 2, Active: 1, Frozen: 0)
Target    :  0.50 0.54 
Activation:  0.51 0.51 
=============================
Layer 'output': (Kind: Output, Size: 2, Active: 1, Frozen: 0)
Target    :  0.00 1.00 
Activation:  0.50 0.46 
=============================
Layer 'hidden': (Kind: Hidden, Size: 20, Active: 1, Frozen: 0)
Activation:  0.00 0.00 0.00 0.00 0.00 0.00 0.01 0.00 0.02 0.00 0.00 0.01 0.00 0.00 0.01 0.01 0.00 0.00 0.00 0.01 
=============================
Layer 'input': (Kind: Input, Size: 6, Active: 1, Frozen: 0)
Activation:  0.00 1.00 1.00 1.00 1.00 0.00 
<q>uit, <g>o, or press Enter: 
-----------------------------------Pattern # 53
Display network 'Backprop Network':
=============================
Layer 'hints': (Kind: Output, Size: 2, Active: 1, Frozen: 0)
Target    :  0.54 0.48 
Activation:  0.51 0.51 
=============================
Layer 'output': (Kind: Output, Size: 2, Active: 1, Frozen: 0)
Target    :  1.00 0.00 
Activation:  0.46 0.48 
=============================
Layer 'hidden': (Kind: Hidden, Size: 20, Active: 1, Frozen: 0)
Activation:  0.04 0.04 0.04 0.04 0.04 0.04 0.05 0.04 0.02 0.04 0.04 0.04 0.04 0.04 0.04 0.04 0.04 0.04 0.04 0.05 
=============================
Layer 'input': (Kind: Input, Size: 6, Active: 1, Frozen: 0)
Activation:  0.00 0.00 0.00 0.00 0.00 1.00 
<q>uit, <g>o, or press Enter: 
-----------------------------------Pattern # 52
Display network 'Backprop Network':
=============================
Layer 'hints': (Kind: Output, Size: 2, Active: 1, Frozen: 0)
Target    :  0.50 0.32 
Activation:  0.51 0.47 
=============================
Layer 'output': (Kind: Output, Size: 2, Active: 1, Frozen: 0)
Target    :  0.00 0.00 
Activation:  0.50 0.32 
=============================
Layer 'hidden': (Kind: Hidden, Size: 20, Active: 1, Frozen: 0)
Activation:  0.00 0.00 0.00 0.00 0.00 0.00 0.01 0.00 0.29 0.00 0.00 0.01 0.00 0.00 0.01 0.01 0.00 0.00 0.00 0.01 
=============================
Layer 'input': (Kind: Input, Size: 6, Active: 1, Frozen: 0)
Activation:  1.00 1.00 1.00 1.00 0.00 0.00 
<q>uit, <g>o, or press Enter: 
-----------------------------------Pattern # 19
Display network 'Backprop Network':
=============================
Layer 'hints': (Kind: Output, Size: 2, Active: 1, Frozen: 0)
Target    :  0.52 0.58 
Activation:  0.51 0.49 
=============================
Layer 'output': (Kind: Output, Size: 2, Active: 1, Frozen: 0)
Target    :  1.00 1.00 
Activation:  0.48 0.42 
=============================
Layer 'hidden': (Kind: Hidden, Size: 20, Active: 1, Frozen: 0)
Activation:  0.00 0.00 0.00 0.01 0.00 0.01 0.02 0.00 0.08 0.00 0.01 0.01 0.00 0.00 0.01 0.01 0.00 0.00 0.00 0.02 
=============================
Layer 'input': (Kind: Input, Size: 6, Active: 1, Frozen: 0)
Activation:  1.00 1.00 0.00 1.00 1.00 0.00 
<q>uit, <g>o, or press Enter: 
-----------------------------------Pattern # 38
Display network 'Backprop Network':
=============================
Layer 'hints': (Kind: Output, Size: 2, Active: 1, Frozen: 0)
Target    :  0.50 0.46 
Activation:  0.51 0.50 
=============================
Layer 'output': (Kind: Output, Size: 2, Active: 1, Frozen: 0)
Target    :  0.00 0.00 
Activation:  0.50 0.46 
=============================
Layer 'hidden': (Kind: Hidden, Size: 20, Active: 1, Frozen: 0)
Activation:  0.01 0.01 0.01 0.01 0.01 0.01 0.02 0.01 0.04 0.01 0.01 0.01 0.01 0.01 0.01 0.02 0.01 0.01 0.01 0.02 
=============================
Layer 'input': (Kind: Input, Size: 6, Active: 1, Frozen: 0)
Activation:  1.00 0.00 1.00 0.00 1.00 0.00 
<q>uit, <g>o, or press Enter: 
-----------------------------------Pattern # 49
Display network 'Backprop Network':
=============================
Layer 'hints': (Kind: Output, Size: 2, Active: 1, Frozen: 0)
Target    :  0.53 0.72 
Activation:  0.51 0.46 
=============================
Layer 'output': (Kind: Output, Size: 2, Active: 1, Frozen: 0)
Target    :  1.00 1.00 
Activation:  0.47 0.28 
=============================
Layer 'hidden': (Kind: Hidden, Size: 20, Active: 1, Frozen: 0)
Activation:  0.04 0.05 0.04 0.04 0.04 0.04 0.06 0.04 0.34 0.04 0.04 0.05 0.04 0.04 0.05 0.05 0.04 0.04 0.04 0.06 
=============================
Layer 'input': (Kind: Input, Size: 6, Active: 1, Frozen: 0)
Activation:  1.00 0.00 0.00 0.00 0.00 0.00 
<q>uit, <g>o, or press Enter: 
-----------------------------------Pattern # 17
Display network 'Backprop Network':
=============================
Layer 'hints': (Kind: Output, Size: 2, Active: 1, Frozen: 0)
Target    :  0.47 0.62 
Activation:  0.51 0.49 
=============================
Layer 'output': (Kind: Output, Size: 2, Active: 1, Frozen: 0)
Target    :  0.00 1.00 
Activation:  0.47 0.38 
=============================
Layer 'hidden': (Kind: Hidden, Size: 20, Active: 1, Frozen: 0)
Activation:  0.03 0.02 0.05 0.04 0.04 0.06 0.07 0.03 0.19 0.05 0.06 0.05 0.03 0.03 0.06 0.04 0.04 0.04 0.03 0.09 
=============================
Layer 'input': (Kind: Input, Size: 6, Active: 1, Frozen: 0)
Activation:  0.00 1.00 0.00 0.00 0.00 0.00 
<q>uit, <g>o, or press Enter: 
-----------------------------------Pattern # 46
Display network 'Backprop Network':
=============================
Layer 'hints': (Kind: Output, Size: 2, Active: 1, Frozen: 0)
Target    :  0.52 0.50 
Activation:  0.51 0.52 
=============================
Layer 'output': (Kind: Output, Size: 2, Active: 1, Frozen: 0)
Target    :  1.00 1.00 
Activation:  0.48 0.50 
=============================
Layer 'hidden': (Kind: Hidden, Size: 20, Active: 1, Frozen: 0)
Activation:  0.02 0.01 0.02 0.02 0.01 0.02 0.03 0.02 0.02 0.02 0.02 0.02 0.01 0.02 0.02 0.03 0.02 0.02 0.01 0.03 
=============================
Layer 'input': (Kind: Input, Size: 6, Active: 1, Frozen: 0)
Activation:  0.00 0.00 0.00 1.00 1.00 0.00 
<q>uit, <g>o, or press Enter: 
-----------------------------------Pattern # 6
Display network 'Backprop Network':
=============================
Layer 'hints': (Kind: Output, Size: 2, Active: 1, Frozen: 0)
Target    :  0.52 0.49 
Activation:  0.51 0.52 
=============================
Layer 'output': (Kind: Output, Size: 2, Active: 1, Frozen: 0)
Target    :  1.00 1.00 
Activation:  0.48 0.51 
=============================
Layer 'hidden': (Kind: Hidden, Size: 20, Active: 1, Frozen: 0)
Activation:  0.04 0.03 0.04 0.05 0.04 0.05 0.06 0.04 0.02 0.04 0.05 0.05 0.03 0.03 0.05 0.04 0.04 0.04 0.03 0.07 
=============================
Layer 'input': (Kind: Input, Size: 6, Active: 1, Frozen: 0)
Activation:  0.00 0.00 0.00 0.00 1.00 0.00 
<q>uit, <g>o, or press Enter: 
-----------------------------------Pattern # 60
Display network 'Backprop Network':
=============================
Layer 'hints': (Kind: Output, Size: 2, Active: 1, Frozen: 0)
Target    :  0.49 0.46 
Activation:  0.51 0.52 
=============================
Layer 'output': (Kind: Output, Size: 2, Active: 1, Frozen: 0)
Target    :  1.00 1.00 
Activation:  0.51 0.54 
=============================
Layer 'hidden': (Kind: Hidden, Size: 20, Active: 1, Frozen: 0)
Activation:  0.02 0.01 0.02 0.02 0.02 0.02 0.03 0.02 0.00 0.02 0.02 0.02 0.01 0.01 0.02 0.02 0.02 0.02 0.01 0.04 
=============================
Layer 'input': (Kind: Input, Size: 6, Active: 1, Frozen: 0)
Activation:  0.00 0.00 0.00 0.00 1.00 1.00 
<q>uit, <g>o, or press Enter: 
-----------------------------------Pattern # 7
Display network 'Backprop Network':
=============================
Layer 'hints': (Kind: Output, Size: 2, Active: 1, Frozen: 0)
Target    :  0.52 0.55 
Activation:  0.51 0.51 
=============================
Layer 'output': (Kind: Output, Size: 2, Active: 1, Frozen: 0)
Target    :  0.00 0.00 
Activation:  0.52 0.55 
=============================
Layer 'hidden': (Kind: Hidden, Size: 20, Active: 1, Frozen: 0)
Activation:  0.02 0.01 0.02 0.02 0.02 0.03 0.04 0.01 0.02 0.02 0.03 0.02 0.01 0.01 0.03 0.02 0.02 0.02 0.01 0.05 
=============================
Layer 'input': (Kind: Input, Size: 6, Active: 1, Frozen: 0)
Activation:  0.00 1.00 0.00 0.00 0.00 1.00 
<q>uit, <g>o, or press Enter: 
-----------------------------------Pattern # 50
Display network 'Backprop Network':
=============================
Layer 'hints': (Kind: Output, Size: 2, Active: 1, Frozen: 0)
Target    :  0.48 0.54 
Activation:  0.51 0.52 
=============================
Layer 'output': (Kind: Output, Size: 2, Active: 1, Frozen: 0)
Target    :  1.00 0.00 
Activation:  0.52 0.54 
=============================
Layer 'hidden': (Kind: Hidden, Size: 20, Active: 1, Frozen: 0)
Activation:  0.00 0.00 0.00 0.00 0.00 0.00 0.01 0.00 0.01 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.01 
=============================
Layer 'input': (Kind: Input, Size: 6, Active: 1, Frozen: 0)
Activation:  1.00 1.00 0.00 1.00 1.00 1.00 
<q>uit, <g>o, or press Enter: 
-----------------------------------Pattern # 21
Display network 'Backprop Network':
=============================
Layer 'hints': (Kind: Output, Size: 2, Active: 1, Frozen: 0)
Target    :  0.46 0.52 
Activation:  0.51 0.52 
=============================
Layer 'output': (Kind: Output, Size: 2, Active: 1, Frozen: 0)
Target    :  1.00 0.00 
Activation:  0.54 0.52 
=============================
Layer 'hidden': (Kind: Hidden, Size: 20, Active: 1, Frozen: 0)
Activation:  0.00 0.00 0.00 0.01 0.00 0.00 0.01 0.00 0.00 0.00 0.00 0.01 0.00 0.00 0.01 0.01 0.00 0.00 0.00 0.01 
=============================
Layer 'input': (Kind: Input, Size: 6, Active: 1, Frozen: 0)
Activation:  1.00 0.00 0.00 1.00 1.00 1.00 
<q>uit, <g>o, or press Enter: 
-----------------------------------Pattern # 40
Display network 'Backprop Network':
=============================
Layer 'hints': (Kind: Output, Size: 2, Active: 1, Frozen: 0)
Target    :  0.45 0.51 
Activation:  0.51 0.52 
=============================
Layer 'output': (Kind: Output, Size: 2, Active: 1, Frozen: 0)
Target    :  1.00 1.00 
Activation:  0.55 0.49 
=============================
Layer 'hidden': (Kind: Hidden, Size: 20, Active: 1, Frozen: 0)
Activation:  0.01 0.00 0.01 0.01 0.01 0.01 0.02 0.01 0.02 0.01 0.01 0.01 0.00 0.01 0.01 0.01 0.01 0.01 0.00 0.03 
=============================
Layer 'input': (Kind: Input, Size: 6, Active: 1, Frozen: 0)
Activation:  0.00 1.00 0.00 1.00 1.00 0.00 
<q>uit, <g>o, or press Enter: 
-----------------------------------Pattern # 10
Display network 'Backprop Network':
=============================
Layer 'hints': (Kind: Output, Size: 2, Active: 1, Frozen: 0)
Target    :  0.57 0.52 
Activation:  0.50 0.52 
=============================
Layer 'output': (Kind: Output, Size: 2, Active: 1, Frozen: 0)
Target    :  0.00 0.00 
Activation:  0.57 0.52 
=============================
Layer 'hidden': (Kind: Hidden, Size: 20, Active: 1, Frozen: 0)
Activation:  0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 
=============================
Layer 'input': (Kind: Input, Size: 6, Active: 1, Frozen: 0)
Activation:  1.00 1.00 1.00 1.00 1.00 1.00 
<q>uit, <g>o, or press Enter: 
-----------------------------------Pattern # 25
Display network 'Backprop Network':
=============================
Layer 'hints': (Kind: Output, Size: 2, Active: 1, Frozen: 0)
Target    :  0.45 0.52 
Activation:  0.51 0.51 
=============================
Layer 'output': (Kind: Output, Size: 2, Active: 1, Frozen: 0)
Target    :  1.00 1.00 
Activation:  0.55 0.48 
=============================
Layer 'hidden': (Kind: Hidden, Size: 20, Active: 1, Frozen: 0)
Activation:  0.00 0.00 0.00 0.01 0.00 0.01 0.01 0.00 0.05 0.00 0.01 0.01 0.00 0.00 0.01 0.01 0.00 0.00 0.00 0.02 
=============================
Layer 'input': (Kind: Input, Size: 6, Active: 1, Frozen: 0)
Activation:  1.00 1.00 1.00 0.00 1.00 0.00 
<q>uit, <g>o, or press Enter: 
-----------------------------------Pattern # 61
Display network 'Backprop Network':
=============================
Layer 'hints': (Kind: Output, Size: 2, Active: 1, Frozen: 0)
Target    :  0.43 0.53 
Activation:  0.51 0.51 
=============================
Layer 'output': (Kind: Output, Size: 2, Active: 1, Frozen: 0)
Target    :  1.00 1.00 
Activation:  0.57 0.47 
=============================
Layer 'hidden': (Kind: Hidden, Size: 20, Active: 1, Frozen: 0)
Activation:  0.02 0.03 0.02 0.02 0.02 0.01 0.01 0.02 0.09 0.02 0.01 0.02 0.02 0.02 0.02 0.04 0.02 0.02 0.02 0.01 
=============================
Layer 'input': (Kind: Input, Size: 6, Active: 1, Frozen: 0)
Activation:  0.00 0.00 1.00 1.00 0.00 0.00 
<q>uit, <g>o, or press Enter: 
-----------------------------------Pattern # 14
Display network 'Backprop Network':
=============================
Layer 'hints': (Kind: Output, Size: 2, Active: 1, Frozen: 0)
Target    :  0.42 0.48 
Activation:  0.50 0.51 
=============================
Layer 'output': (Kind: Output, Size: 2, Active: 1, Frozen: 0)
Target    :  1.00 0.00 
Activation:  0.58 0.48 
=============================
Layer 'hidden': (Kind: Hidden, Size: 20, Active: 1, Frozen: 0)
Activation:  0.04 0.07 0.04 0.04 0.04 0.03 0.03 0.05 0.09 0.03 0.03 0.04 0.04 0.04 0.04 0.06 0.04 0.04 0.05 0.03 
=============================
Layer 'input': (Kind: Input, Size: 6, Active: 1, Frozen: 0)
Activation:  0.00 0.00 1.00 0.00 0.00 0.00
This is a test.
This is a test.